Allocating storage units in a dispersed storage network

ABSTRACT

A method begins by a processing module detecting a new storage unit within a dispersed storage network DSN. The method continues with the processing module determining to affiliate the new storage unit with a virtual memory vault, where the virtual memory vault is mapped to a set of storage units of the DSN, where data objects are dispersed storage error encoded into pluralities of sets of encoded data slices that are stored in the set of storage units. The method continues with the processing module determining virtual DSN addresses of the virtual memory vault to allocate the new storage unit. The method continues with the processing module updating vault information regarding the virtual memory vault to include the allocation of the virtual DSN addresses to the new storage unit.

CROSS REFERENCE TO RELATED PATENTS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. §120 as a continuation-in-part of U.S. Utility applicationSer. No. 13/594,199, entitled “DISPERSED STORAGE NETWORK VIRTUAL ADDRESSGENERATIONS”, filed Aug. 24, 2012, now U.S. Pat. No. 8,886,905, issuedon Nov. 11, 2014, which is a continuation of U.S. Utility applicationSer. No. 12/764,930, entitled “DISPERSED STORAGE NETWORK VIRTUAL ADDRESSGENERATIONS”, filed Apr. 21, 2010, now U.S. Pat. No. 8,275,966, issuedSep. 25, 2012, which claims priority pursuant to 35 U.S.C. §119(e) toU.S. Provisional Application No. 61/230,019, entitled “DISPERSED STORAGENETWORK VIRTUAL ADDRESS SPACE”, filed Jul. 30, 2009, all of which arehereby incorporated herein by reference in their entirety and made partof the present U.S. Utility patent application for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

This invention relates generally to computing and more particularly tostorage of information.

2. Description of Related Art

Computing systems are known to communicate, process, and store data.Such computing systems range from wireless smart phones to data centersthat support millions of web searches, stock trades, or on-linepurchases every day. Each type of computing system is constructed, andhence operates, in accordance with one or more communication,processing, and storage standards. With such standards, and withadvances in technology, more and more of the global information contentis being converted into electronic formats. Electronic contentpervasiveness is producing increasing demands on the storage function ofcomputing systems.

A typical computer storage function includes one or more memory devicesto match the needs of the various operational aspects of the processingand communication functions. For example, a memory device may includesolid-state NAND flash, random access memory (RAM), read only memory(ROM), a mechanical hard disk drive. Each type of memory device has aparticular performance range and normalized cost. The computing systemarchitecture optimizes the use of one or more types of memory devices toachieve the desired functional and performance goals of the computingsystem. Generally, the immediacy of access and power requirementsdictates what type of memory device is used. For example, RAM memory canbe accessed in any random order with a constant response time, thoughRAM requires continuous power to store data. By contrast, memory devicetechnologies that require physical movement such as magnetic discs,tapes, and optical discs, have a variable responses time as the physicalmovement can take longer than the data transfer, but do not requirecontinuous power to store data.

Each type of computer storage system is constructed, and hence operates,in accordance with one or more storage standards. For instance, computerstorage systems may operate in accordance with one or more standardsincluding, but not limited to network file system (NFS), flash filesystem (FFS), disk file system (DFS), small computer system interface(SCSI), internet small computer system interface (iSCSI), file transferprotocol (FTP), and web-based distributed authoring and versioning(WebDAV). An operating systems (OS) and storage standard may specify thedata storage format and interface between the processing subsystem andthe memory devices. The interface may specify a structure such asdirectories and files. Typically, a memory controller provides aninterface function between the processing function and memory devices.As new storage systems are developed, the memory controller functionalrequirements may change to adapt to new standards.

However, memory devices may fail, especially those that utilizetechnologies that require physical movement like a disc drive. Forexample, it is not uncommon for a disc drive to suffer from bit levelcorruption on a regular basis, or complete drive failure after anaverage of three years of use. One common solution is to utilize morecostly disc drives that have higher quality internal components. Anothersolution is to utilize multiple levels of redundant disc drives to abatethese issues by replicating the data into two or more copies. One suchredundant drive approach is called redundant array of independent discs(RAID). Multiple physical discs comprise an array where parity data isadded to the original data before storing across the array. The parityis calculated such that the failure of one or more discs will not resultin the loss of the original data. The original data can be reconstructedfrom the other discs. RAID 5 uses three or more discs to protect datafrom the failure of any one disc. The parity and redundancy overheadreduces the capacity of what three independent discs can store by onethird (n−1=3−2=2 discs of capacity using 3 discs). RAID 6 can recoverfrom a loss of two discs and requires a minimum of four discs with anefficiency of n−2. Typical RAID systems utilize a RAID control to encodeand decode the data across the array.

Unfortunately, there are several drawbacks of the RAID approach,including effectiveness, efficiency and security. As more discs areadded, the probability of one or two discs failing rises, and is notnegligible, especially if less costly discs are used. Thus, the RAIDapproach has not proved to be effective in maintaining the integrity ofthe data over time. It is also common to mirror RAID arrays at differentphysical locations. For example, if the RAID array is part of a nationallevel computing system with occasional site outages, multiple sites maybe employed, each containing one or more RAID arrays. However, storingredundant copies at remote locations is not an efficient method for datastorage or retrieval. In addition, unauthorized file access becomes amore acute problem when whole copies of the same file are replicated,either on just one storage system site or at two or more sites.

Therefore, a need exists to provide a data storage solution thatprovides more effective timeless continuity of data, minimizes adverseaffects of multiple memory elements failures, provides improvedsecurity, can be adapted to a wide variety storage system standards andis compatible with computing and communications systems.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a computingsystem in accordance with the invention;

FIG. 2 is a diagram of an embodiment of a dispersed storage data slicingin accordance with the present invention;

FIG. 3 is a schematic block diagram of an embodiment of a computing corein accordance with the present invention;

FIG. 4 is a schematic block diagram of an embodiment of a file systemhierarchy in accordance with the present invention;

FIG. 5 a is a diagram of an embodiment of a dispersed storage memorymapping in accordance with the present invention;

FIGS. 5 b-e are a diagram of another embodiment of a dispersed storagememory mapping illustrating an example of evolution of the memorymapping for evolving dispersed storage network (DSN) memory in a DSN inaccordance with the present invention;

FIG. 5 f is a flowchart illustrating an example of mapping a virtualmemory to a set of storage units in accordance with the presentinvention;

FIG. 6 a is a diagram of an embodiment of a dispersed storage addressmapping in accordance with the present invention;

FIG. 6 b is a diagram of another embodiment of a dispersed storageaddress mapping in accordance with the present invention;

FIG. 6 c is a diagram of another embodiment of a dispersed storageaddress mapping in accordance with the present invention;

FIG. 7 is a logic diagram of an embodiment of a method for storage of adata object within a dispersed storage network (DSN) in accordance withthe present invention;

FIG. 8 is a diagram of an embodiment of a dispersed storage addresstable in accordance with the present invention;

FIG. 9 is a schematic block diagram of an embodiment of a dispersedstorage processing unit in accordance with the present invention;

FIG. 10 is a schematic block diagram of an embodiment of a grid modulein accordance with the present invention;

FIG. 11 is a diagram of another embodiment of a dispersed storageaddress table in accordance with the present invention;

FIG. 12 a is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 12 b is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 12 c is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 12 d is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 12 e is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 12 f is a diagram of another embodiment of a dispersed storagememory mapping in accordance with the present invention;

FIG. 13 is a logic diagram of an embodiment of a method for addressassignment in accordance with the present invention;

FIG. 14 is a diagram of another embodiment of a dispersed storageaddress table in accordance with the present invention;

FIG. 15 a is a diagram of another embodiment of a dispersed storageaddress mapping in accordance with the present invention;

FIG. 15 b is a diagram of another embodiment of a dispersed storageaddress mapping in accordance with the present invention;

FIG. 15 c is a diagram of another embodiment of a dispersed storageaddress mapping in accordance with the present invention;

FIG. 16 a is a logic diagram of another embodiment of a method foraddress assignment in accordance with the present invention;

FIG. 16 b is a logic diagram of another embodiment of a method foraddress assignment in accordance with the present invention; and

FIG. 17 is a logic diagram of another embodiment of a method for addressassignment in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of an embodiment of a computingsystem that includes a first user device 10, a dispersed storage (DS)processing unit 12, a dispersed storage network (DSN) memory 14, astorage integrity processing unit 16, a second user device 18, a DSmanaging unit 20, and a network 22 providing connectivity between thedevices 10, 18 and units 12, 16, 24-26. The DSN memory 14 includes aplurality of dispersed storage (DS) storage units 24-26. The DSN memory14 may be a subset of the total set of DS storage units 24-26. In anexample of operation, data from the user devices 10, 18 is stored toand/or retrieved from the DSN memory 14 as will be described in greaterdetail below.

The first user device 10 and the second user device 18 may each be aportable device or a fixed device. For example, a portable device may bea cell phone, a smart phone, a personal digital assistant, a digitalmusic player, a digital video player, a laptop computer, a handheldcomputer, a video game controller, and/or any other portable computingequipment. A fixed device may be a personal computer, a computer server,a cable set-top box, a satellite receiver, a television set, a printer,a fax machine, home entertainment equipment, a video game console,and/or any type of home or office computing equipment.

The DS processing unit 12, one or more of the DS storage units 24-26,the storage integrity processing unit 16, and the DS managing unit 20each may be a portable device or a fixed device or may be a functionalmodule within another unit that is a portable device or a fixed device.For example, the DS managing unit 20 may be a computer server and thestorage integrity processing unit 16 may be a functional moduleoperating on the same computer server as the DS managing unit 20. Inanother example, the DS processing unit 12 may be a functional moduleoperating on one of the user devices 10 or 18 or DS storage units 24-26.

The network 22 may be a wire lined and/or wireless communication systemor system of systems that provide communications capability between thedevices 10, 18 and units 12, 16, 24-26. The system of systems may be acombination of private intranets and the public internet. In an example,the DS processing unit 12 has access to the network 22 by way of anoptical sub-network and the second user device 18 has access to thenetwork 22 by way of a 4G Long Term Evolution (LTE) wireless network.

In an embodiment, the first user device 10 includes a computing core 40and an interface 42 to communicate with the DS processing unit 12. Theinterface 42 accommodates one or more storage system standards such thatthe first user device 10 can utilize a storage standard native to itsoperating system. The computing core 40 may include a processing moduleand other components as will be described in greater detail withreference to FIG. 3.

In an embodiment, the DS processing unit 12 includes an interface 34 tothe first user device 10, and a dispersed storage network (DSnet)interface 36 to communicate in a format compatible with the rest of thecomputing system. For example, the DSnet interface 36 will transmit datato one or more of the DS storage units 24-26 in a format compatible tostore the data on the DS storage units 24-26 using a virtual addressingscheme of the DSN memory 14. The virtual addressing scheme of the DSNmemory 14 utilizes a virtual address for a data object 48 that is storedto and retrieved from the dispersed storage network. The DSnet interface36 will be described in greater detail with reference to FIGS. 4 and 9.The interface 34 to the first user device 10 accommodates one or morestorage system standards such that the first user device 10, or otheruser devices, can utilize a storage standard native to their operatingsystem. A user ID 44 and an OS filename or object name 46 are includedin the communications where the OS filename 46 is compatible with the OSof the user device 10, 18. The OS filename 46 identifies the data object48 that is stored to and retrieved from the dispersed storage network.

In an embodiment, the second user device 18 includes a computing core 28and a DSnet interface 32 to communicate in a format compatible with therest of the computing system. For example, the DSnet interface 32 willtransmit data to one or more of the DS storage units 24-26 in a formatcompatible to store the data on the DS storage units 24-26. Thecomputing core 28 may include a DS processing module 30 and othercomponents as will be described in greater detail with reference to FIG.3. The DS processing module 30 will perform a similar function as the DSprocessing unit 12, thus eliminating the need for an external DSprocessing unit 12 to the second user device 18. In other words, the DSprocessing module 30 of the second user device 18 enables the seconduser device 18 to directly store data to and/or retrieve data from theDSN memory 14 utilizing the virtual addressing scheme of the DSN memory14.

In an embodiment, the storage integrity processing unit 16 includes aDSnet interface 38 to communicate in a format compatible with the restof the computing system. For example, the DSnet interface 38 willreceive data from one or more of the DS storage units 24-26 in a formatcompatible to decode the data from the DS storage units 24-26. In anexample of operation, the storage integrity processing unit 16 scans thecontents of the DSN memory 14 to detect undesired conditions includingdata corruption, missing data, and offline DS storage units 24-26. Thestorage integrity processing unit 16 will rebuild a portion of the datain the DSN memory 14 and alert the DS managing unit 20 when undesiredconditions are detected. The storage integrity processing unit 16rebuilds the data by retrieving available data, processing the availabledata to produce rebuilt data, and storing the rebuilt data in the DSNmemory 14.

The DS managing unit 20 functions may include receiving and aggregatingnetwork management alarms, alerts, errors, status information,performance information, and messages from any of the modules and units10, 12, 14, 16, 18, 24-26 of the computing system. For example, the DSprocessing unit 12 may transmit a simple network management protocol(SNMP) message regarding the status of the DS processing unit 12 via thenetwork 22 to the DS managing unit 20. The DS managing unit 20 functionsmay also include functions to configure the computing system and performa billing function for the computing system. For example, the DSmanaging unit 20 may determine the number of DS storage units 24-26 toconfigure to meet the operation requirements of a particular user. Theconfiguration may include assignment of DSN memory addresses. In anotherexample, the DS managing unit 20 may track the usage of the DSN memory14 by the user to create a summary and/or bill. The DS managing unit 20may also automatically determine optimization of the configuration ofthe computing system based in part from determining the correlation ofpast and present configurations with performance. The DS managing unit20 may share the correlation and configurations with other computingsystems managing a different DSN memory to further optimize thecomputing system.

In an example of operation of the computing system, the computing core40 of the first user device 10 initiates the sequence to store data tothe DSN memory 14 by transmitting the data object 48 via the first userdevice interface 42 to the DS processing unit interface 34. The dataobject 48 may include a data file, data blocks, a real time data streamor any other format of digital information. The protocol may include theuser ID 44 and the OS filename or object name 46 to mimic a conventionalfile system interface and may be any one or more of network file system(NFS), flash file system (FFS), disk file system (DFS), small computersystem interface (SCSI), internet small computer system interface(iSCSI), file transfer protocol (FTP), and web-based distributedauthoring and versioning (WebDAV).

The DS processing unit 12 receives the data and processes it to createsets of error coded (EC) data slices 49. For example, as shown in FIG.2, the data, or data object 48, is broken down into Y data segments 47,error coded, and sliced into X (e.g., X=16 slices wide) slices persegment, thus creating XY total error coded data slices 49. The numberof slices X per segment is also called the number of pillars and ischosen as a function of the error coding objectives. More specifically,the set of nth slices of all sources in a vault is called a pillar,where a “vault” is associated with one or more users. For example,pillar 3 in a particular vault contains slice 3 of all files written tothis particular vault. The size of the data segments 47 is a function ofwhat is chosen for Y and the size of the data object 48. The size ofeach error coded data slice 49 is a function of the size of the datasegment 47 and the error coding algorithm.

Referring again to FIG. 1, the DS processing unit 12 creates a uniqueslice name 45 for each EC data slice 49 and attaches the slice name 45to each EC data slice 49. The slice name 45 includes universal DSNmemory 14 addressing routing information (i.e., a virtual DSN memoryaddress) and user file information based on the OS filename 46 and userID 44 to tie the two together. The creation and utilization of the slicename 45 will be described in greater detail with reference to FIGS.3-17.

The DS processing unit 12 transmits the EC slices 1 through X, andattached slice names, for each data segment via the DSnet interface 36and network 22 to the DSN memory 14 for storage. The DS storage units24-26 translate the virtual DSN memory address of the slice name 45 intoa local physical address for storage. In an embodiment, each EC slice49, for the same data segment, is stored on a different DS storage unit24-26 from all the other slices of that data segment to improve dataintegrity. The process concludes when the EC slices 49 for the last datasegment Y are stored in the DSN memory 14.

In another example of operation of the computing system, the computingcore 40 of the second user device 18 initiates a data retrieval sequenceby activating the DS processing module 30 to transmit retrieval messagesto the DSN memory 14 via the DSnet interface 32 and network 22. Theretrieval messages are received by the DS storage units 24-26 thatcontain the EC data slices 49 generated from the original data object 48during the storage process. The DS storage units 24-26 read theavailable EC data slices 49, generated from the original data object 48,from the internal memory devices of the DS storage units 24-26 andtransmit the EC slices 49 via the network 22 and DSnet interface 32 ofthe second user device 18 to the DS processing module 30. The DSprocessing module 30 decodes the EC slices 49 in accordance with aninformation dispersal algorithm (IDA) to produce the original dataobject 48.

The above example retrieval sequence would be similar if the first userdevice 10 initiates the retrieval request. In such an example, thecomputing core 40 of the first user device 10 requests the data object48 retrieval using a standard memory access protocol associated with theOS of the first user device 10 utilizing the OS filename 46. The DSprocessing unit 12 performs the same tasks as in the above describedexample to retrieve the EC slices from the appropriate DS storage units24-26 to reconstruct the data object 48 and provide it to the first userdevice 10 via the interfaces 34 and 42 to the computing core 40 of thefirst user device 10.

FIG. 3 is a schematic block diagram of an embodiment of a computing core50 that includes a processing module 52, a memory controller 54, a videographics processing unit 56, a main memory 58, an input output (IO)controller 60, an input output (IO) interface 62, an input output (IO)device interface module 64, a read only memory basic input output system(ROM BIOS) 66, a peripheral component interconnect (PCI) interface 68, auniversal serial bus (USB) interface module 70, a host bus adapter (HBA)interface module 72, a network interface module 74, a flash interfacemodule 76, a hard drive interface module 78, and a DSN interface module80.

The computing core 50 may be included in the user devices, the DSprocessing unit, the DS storage unit, the storage integrity processingunit, the DS managing unit, and/or any other device or unit that is partof or coupled to the computing system of FIG. 1. Some computing coreelements may be external to the computing core 50. For example, thevideo graphics processing unit 56 may be a separate computer card withspecialty integrated circuits to render display images. The videographics processing unit 56 may interface to the computing core 50utilizing a standard such as accelerated graphics port (AGP) orperipheral component interconnect express (PCIe).

The DSN interface module 80 function may be all or partially a part ofthe computing core 50. For example, a portion of the DSN memoryinterface module 80 function may be part of the computing core 50 of theuser device and part of the computing core 50 of the DS processing unitof FIG. 1.

Computing core 50 modules, interfaces, and controllers 52-80 may eachinclude one or more integrated circuits to perform their function aloneor in combination with executable computer instructions. The executablecomputer instructions may be programmed into the integrated circuit orloaded into a memory associated with the computing core 50 for executionby the processing module 52. For example, the ROM BIOS 66 may containexecutable computer instructions that are loaded into the main memory 58and executed by the processing module 52 upon initial start of thecomputing core 50.

In an embodiment, the processing module 52 is the primary processor forthe computing core 50 coordinating a majority of tasks (e.g., executeone or more operation instructions of an algorithm, perform aninterrupt, perform a co-processing function, process data, store data tomemory, read data from memory, etc.) and includes at least one L2 cache82, at least one L1 cache 84, and registers 86. The memory controller 54coordinates the reading of data and writing of data within the coreprocessing unit. For example, data for display renderings may be routedto and from the video graphics processing unit 56 while data for routineoperations may be routed to and from the main memory 58. The videographics processing unit 56 may include a video graphics memory 88.

The IO controller 60 provides access to the memory controller 54 fortypically slower devices. In an embodiment, the IO controller 60, viathe PCI interface 68, provides functionality between the memorycontroller 54 and the DSN memory utilizing an electrical connection anda protocol standard. For example, a local area network (LAN) and/or widearea network (WAN) may interface with the network interface module 80utilizing a network file system (NFS) standard. A flash memory mayinterface with the flash interface module 76 utilizing a flash filesystem (FFS) standard. A hard drive may interface with the HD interfacemodule 78 utilizing a disk file system (DFS). The DSN memory mayinterface with the DSN interface module 80 utilizing a series ofstandards including NFS, FFS, DFS, and more. For example, the DSNinterface module of the first user device of FIG. 1 may utilize NFS tocommunicate data objects to/from the DSN memory. In another example, theDSN interface module of the second user device of FIG. 1 may communicateerror coded data slices to/from the DSN memory.

Portions of the interface modules 70-80 may have capabilities such thatthey can be directly coupled to the IO controller 60 or directly to thememory controller 54. For example, the DSN interface module 80 may bedirectly coupled to the memory controller 54.

The IO interface 62 couples the memory controller 54, via the IOcontroller 60, to other computing core elements including the ROM BIOS66 and the IO device interface module 64. The IO device interface module64 may provide the interface for a variety of IO devices including akeyboard, a mouse, a printer, a removable CD/DVD disk drive, and/or anyother IO device.

In an example of operation, the processing module 52 retrieves data(e.g., a media file) from the main memory 58 routing it through thememory controller 54, the IO controller 60, the PCI interface 68, andthe DSN interface module 80 to the DSN memory for storage.

FIG. 4 is a schematic block diagram of an embodiment of a file systemhierarchy including a plurality of user virtual memories 90-92, aSegment and Slice directory 94, a Pillar to DSN Storage Unit table 96,and a Physical DSN memory 98. The Segment and Slice directory 94comprises a Virtual DSN address space 100, while the physical DSN memory98 includes a plurality of DS storage units A-D. The Segment and Slicedirectory 94 and Pillar to DSN Storage Unit table 96 may be implementedwithin, for example, the DS processing unit and/or DS processing moduleof FIG. 1. The file system hierarchy may be utilized to translate theuser virtual memory 90-92 to the Physical DSN memory 98 by translating adata object associated with a user into the virtual DSN address space100 and then to the Physical DSN memory 98.

Each user is associated with one or more vaults and each vault isassociated with one or more users. The vault may contain per vaultand/or per source of the vault, user attributes (e.g., useridentification, billing data, etc.), operational parameters, a list ofthe DS storage units that will be utilized to support the vault and alist of virtual memory addresses within the Virtual DSN address space100 assigned to the vault. One or more vaults may be established fromthe same set of DS storage units in different combinations. For example,User #1's vault can utilize DS storage units A, B, C and D, while User#N's vault can utilize DS storage units A, C and D. A vault identifieris utilized to distinguish between vaults. A vault has a predeterminednumber of pillars for each data object. For example, User #1 has a vaultidentified operational parameter of four pillars (Pillars 0-3).

Although pillars can be allocated to more than one DS storage unit, inone embodiment, a given DS storage unit is not assigned to store morethan one pillar from a given vault to improve system robustness.Therefore, within the Pillar to DSN Storage Unit table 96, each Pillar(0-3) is assigned to a different DS storage unit (A-D), respectively.However, in other embodiments, instead of assigning pillars to differentDS storage units (A-D), the Pillar to DSN Storage Unit table 96 canindicate that a given DS storage unit will only store one slice of anygiven source, but may store multiple slices from the same pillar. Forexample, DS Storage Unit A may store pillar 1 of sources 51-100 andpillar 2 of sources 0-50. As such, the DSN to Storage Unit table 96 mayallow arbitrary name range mappings rather than requiring alignment onpillar boundaries. The vault can be maintained by, for example, the DSmanaging unit of FIG. 1 and can be used to populate the Segment andSlice directory 94 and Pillar to DSN storage unit table 96.

In an embodiment, one of the plurality of users (i.e., User #1) utilizesa native OS file system to access User #1's virtual memory 90 byincluding source name information (i.e., a file/block name identifying adata object and a vault identifier associated with the user) in anyrequest such as read, write, delete, list, etc. The source name vaultidentifier and file/block name will index into the Segment and Slicedirectory 94 to determine the Virtual DSN address space 100 assigned tothat data object. For example, each data slice of the data object isassigned a unique virtual DSN address within the Virtual DSN addressspace 100 allocated to User #1's vault.

In an example embodiment, the total virtual DSN address space 100 isdefined by a forty eight byte identifier thus creating 256⁴⁸ possibleslice names. The virtual DSN address space 100 accommodates addressingall EC (error coded) data slices of every data segment of every dataobject (e.g., data file, blocks, streams) of every user vault. The slicename is a virtual DSN address and remains the same even as different DSstorage units are added or deleted from the physical DSN memory 98.

As described above, a user has a range of virtual DSN address assignedto their vault. In one embodiment, the virtual DSN addresses typicallydo not change over the operational lifespan of the system for thatparticular user. In another embodiment, the virtual DSN address space100 is dynamically altered from time to time to provide potentialbenefits including improved security and expansion, or retraction,capability. A virtual DSN address space security algorithm may alter thevirtual DSN address space 100 according to one or more of a command(e.g., from the DS managing unit), a schedule, a detected securitybreach, or any other trigger. The virtual DSN address may also beencrypted in use thus requiring encryption and decryption steps wheneverthe virtual DSN address is used.

The index into the virtual DSN address space 100 by vault and file namewill create the slice names (virtual DSN addresses) to use as an indexinto the Pillar to DSN Storage Unit table 96. In an embodiment, thePillar to DSN Storage Unit table 96 is sorted by vaults and pillars sothat the subsequent addresses are organized by pillar of all the filesegments that have EC data slices with the same slice identifier andhence are typically stored at the same DS storage unit. The output ofthe access to the Pillar to DSN Storage Unit table 96 is the DS storageunit identifiers.

The slice names will be used as the virtual index to the memory systemof each DS storage unit to gain access the physical address of the ECdata slices. In an embodiment, the DS storage unit maintains a localtable (not shown) correlating slice names (virtual DSN address) to theaddresses of the physical media internal to the DS storage unit.

In the example shown in FIG. 4, User #1 has a vault identifiedoperational parameter of four pillars, and as such, there are four dataslices created for each data segment of the Y segments. Pillar 0 ismapped to DS storage unit A, pillar 1 is mapped to DS storage unit B,pillar 2 is mapped to DS storage unit C, and pillar 3 is mapped to DSstorage unit D.

FIG. 5 a is a diagram of an embodiment of a dispersed storage memorymapping including slice name ranges 97 that are mapped to particular DSstorage unit identifiers. This mapping indicates the Pillar to DSNStorage Unit table of FIG. 4. Each range 97 has a common sliceidentifier, hence that range is consistent with a single pillar. Toprovide desired error resiliency, the same pillar may utilize one ormore DS storage units, but at most one pillar (of a given vault) willutilize the same DS storage unit. In other words, pillars of the samevault utilize different DS storage units. In an example, pillar 0 ofvault λ utilizes DS storage unit A and pillar 3 of vault λ utilizes DSstorage unit H. In another example, pillar 0 of vault θ utilizes DSstorage units A and B while pillar 3 of vault θ utilizes DS storage unitH.

The slice name range 97 is derived from common slice identifiersdetermined from the slice name 45. The common slice identifiers from theslice name 45 include the vault identifier 55, the slice index 65 andthe vault generation number 75 (e.g., which indicates a particulargeneration of a vault, as described in more detail below in connectionwith FIGS. 14-17). In general, a new generation for a vault isestablished in order to provide additional storage capacity for aparticular user. Other elements 85 of the slice name 45 that do notindicate a particular pillar, but rather a particular data slice,include the file name/block identifier and the data segment number.Slice index numbers on the boundaries of a split between two or more DSstorage units may be random or chosen to meet a split criteria. Forexample, the split criteria may force the boundaries to be between filenumbers, segment numbers of the same file number, generations of thesame file, or other (e.g., reserved bits, flags, future expansion).

In an example, the slice name field comprises forty-eight bytesincluding sixteen bytes for the vault identifier 55, two bytes for theslice index 65, two bytes for the vault generation number 75, two bytesfor a type identifier (e.g., file or block), sixteen bytes for a fileidentifier number (or fourteen bytes of zeros when blocks are used),four bytes for a reserved field, and six bytes for the data segmentnumber. In one embodiment, slice names for pillars are specified inranges rather than byte by byte. The slice name ranges may be broken upinto any number of ranges where the number of ranges is less than thenumber of permutations of the address space of the other elements.

FIGS. 5 b-e are diagrams of other embodiments of a dispersed storagememory mapping illustrating examples of evolution of the memory mappingfor evolving dispersed storage network (DSN) memory in a DSN. Themapping includes a virtual memory mapping of ranges of slice names 45 todispersed storage (DS) units of a set of DS units. The ranges of slicenames 45 includes virtual DSN address boundaries associated with variousfields of the slice names 45. The various fields of the slice name 45includes a vault ID field, a slice index field, a generation field, anobject number field, and a segment field. For example, the vault IDfield includes an entry of 457, the slice index field includes an entryof 1 for a first pillar or slice number of a set of encoded data slices,the generation field includes an entry of 1 for a first generation ofstorage and an entry of 2 for a second generation of storage, objectnumber entries from 1 to N for the object number field of eachgeneration, and segment entries of 1-Y for the segment field for eachobject number.

Each of a plurality of data objects are stored in the set of DS units asa plurality of sets of encoded data slices. Each plurality of sets ofencoded data slices is associated with a plurality of sets of slicenames. Each slice name associated with a common data object includes acommon vault ID entry, a common generation entry, and a common objectnumber. Each set of encoded data slices associated with a correspondingset of slice names, where each slice name of the set of slice namesincludes a unique slice index entry and a common segment entry. Each setof slice names includes a unique segment number.

In an embodiment, a processing module (e.g., of the DS managing unit 20of FIG. 1), as the memory evolves, generates tracking information bytracking evolutionary change of storage units of the DSN memory (e.g.,change of memory capacity and/or type of memory, addition of a newstorage unit to the DSN memory, deletion of one of the storage units,change in physical memory of one or more storage units), performance andreliability requirements of the DSN memory (e.g., to meet performancegoals), and memory utilization of the DSN memory. The memory utilizationof the DSN memory includes one or more of creation of a new virtualmemory vault, modification of the virtual memory vault, deletion of anexisting virtual memory vault, data retrieval frequencies for thevirtual memory vault, data storage rates for the virtual memory vault,per storage unit available memory, and per data object data size.

For a given state of evolution of the DSN memory, the processing moduleinterprets the tracking information for the given state of evolution toproduce given state DSN memory data. Having produced the given state DSNmemory data, the processing module establishes the virtual DSN addressboundaries for the set of the storage units for storing data objects ina virtual memory vault of the DSN based on the given state DSN memorydata, where the virtual memory vault is mapped to the set of the storageunits, and where the data objects are dispersed storage error encodedinto pluralities of sets of encoded data slices that are stored in theset of the storage units in accordance with the virtual DSN addressboundaries.

FIG. 5 b illustrates an example where the processing module establishesinitial virtual DSN address boundaries at time t0 by slice index where aDS unit 1 is assigned to all address ranges associated with slice index1 and a DS unit 2 is assigned to all address ranges associated withslice index 2 etc. Alternatively, the processing module establishes theinitial virtual DSN address boundaries such that the DS unit 1 isassociated with boundaries of the generation 1 of slice index 1 and theDS unit 2 is associated with boundaries of the generation 1 of sliceindex 2.

From time to time, the processing module may reevaluate to re-establishthe virtual DSN address boundaries. For example, the processing moduledetects a change in the given state of evolution of the DSN memory to anew given state when a change occurs to the virtual memory vault. Forthe new given state, the processing module interprets the trackinginformation for the new given state of evolution to produce new givenstate DSN memory data and establishes new virtual DSN address boundariesfor the set of the storage units based on the new given state DSN memorydata. The establishing of the new virtual DSN address boundaries will bediscussed in greater detail with reference to FIGS. 5 c-e.

As another example, the processing module detects a change in the givenstate of evolution of the DSN memory to a new given state when a changeoccurs to one of the storage units or when a new storage unit is addedto the DSN memory. Having detecting the change in the given state, theprocessing module, for the new given state, interprets the trackinginformation for the new given state of evolution to produce new givenstate DSN memory data, updates the set of storage units to produce anupdated set of storage units, and establishes new virtual DSN addressboundaries for the updated set of the storage units based on the newgiven state DSN memory data.

FIG. 5 c illustrates an example of establishing the virtual DSN addressboundaries as the DSN memory evolves at time t1. As a specific example,the processing module establishes the virtual DSN address boundaries inaccordance with generations of the virtual memory vault, where, for afirst generation of the virtual memory vault, a first storage unit ofthe set of the storage units stores an encoded data slice for each setof encoded data slices of the plurality of sets of encoded data slicesgenerated during the first generation of the virtual memory vault andwhere, for a second generation of the virtual memory vault, a secondstorage unit of the set of the storage units stores an encoded dataslice for each set of encoded data slices of the plurality of sets ofencoded data slices generated during the second generation of thevirtual memory vault. For instance, generation 1 of slice index 1 ismapped to the DS unit 1, generation 2 of slice index 1 is mapped to a DSunit 1a, generation 1 of slice index 2 is mapped to the DS unit 2,generation 2 of slice index 2 is mapped to a DS unit 2a.

FIG. 5 d illustrates an example of establishing the virtual DSN addressboundaries as the DSN memory evolves at time t2. As a specific example,the processing module establishes the virtual DSN address boundaries inaccordance with boundaries of data objects, where, for a first set ofthe data objects, a first storage unit of the set of the storage unitsstores an encoded data slice for each set of encoded data slices of theplurality of sets of encoded data slices corresponding to the first setof the data objects and where, for a second set of the data objects, asecond storage unit of the set of the storage units stores an encodeddata slice for each set of encoded data slices of the plurality of setsof encoded data slices corresponding to the second set of the dataobjects. For instance, data object 1 of generation 1 of slice index 1 ismapped to the DS unit 1, other data objects (e.g., data objects 2−N) ofgeneration 1 of slice index 1 are mapped to a DS unit 1a, data object 1of generation 2 of slice index 1 is mapped to the DS unit 1a, other dataobjects (e.g., data objects 2−N) of generation 2 of slice index 1 aremapped to a DS unit 1c, etc.

FIG. 5 e illustrates an example of establishing the virtual DSN addressboundaries as the DSN memory evolves at time t3. As a specific example,the processing module establishes the virtual DSN address boundaries inaccordance with boundaries of data segments of the data objects, where,for a first set of the data segments, a first storage unit of the set ofthe storage units stores an encoded data slice for each set of encodeddata slices of the plurality of sets of encoded data slicescorresponding to the first set of the data segments and where, for asecond set of the data segments, a second storage unit of the set of thestorage units stores an encoded data slice for each set of encoded dataslices of the plurality of sets of encoded data slices corresponding tothe second set of the data segments. For instance, data segments 1-10 ofdata object 1 of generation 1 of slice index 1 are mapped to the DS unit1, data segments 11-Y of data object 1 of generation 1 of slice index 1are mapped to a DS unit 1d, data segments 1-5 of data object N ofgeneration 1 of slice index 1 are mapped to the DS unit 1b, datasegments 6-Y of data object N of generation 1 of slice index 1 aremapped to a DS unit 1e, etc.

FIG. 5 f is a flowchart illustrating an example of mapping a virtualmemory to a set of storage units of an evolving dispersed storagenetwork (DSN) memory in a DSN. As the DSN memory evolves, the methodcontinues at step 200 where a processing module generates trackinginformation by tracking evolutionary change of storage units of the DSNmemory, performance and reliability requirements of the DSN memory, andmemory utilization of the DSN memory. For a given state of evolution ofthe DSN memory, the method continues at step 202 where the processingmodule interprets the tracking information for the given state ofevolution to produce given state DSN memory data.

The method continues at step 204 where the processing module establishesvirtual DSN address boundaries for the set of the storage units forstoring data objects in a virtual memory vault of the DSN based on thegiven state DSN memory data, where the virtual memory vault is mapped tothe set of the storage units, and where the data objects are dispersedstorage error encoded into pluralities of sets of encoded data slicesthat are stored in the set of the storage units in accordance with thevirtual DSN address boundaries.

The establishing the virtual DSN address boundaries includes severalapproaches. As a specific example, the processing module establishes thevirtual DSN address boundaries in accordance with generations of thevirtual memory vault, where, for a first generation of the virtualmemory vault, a first storage unit of the set of the storage unitsstores an encoded data slice for each set of encoded data slices of theplurality of sets of encoded data slices generated during the firstgeneration of the virtual memory vault and where, for a secondgeneration of the virtual memory vault, a second storage unit of the setof the storage units stores an encoded data slice for each set ofencoded data slices of the plurality of sets of encoded data slicesgenerated during the second generation of the virtual memory vault.

As another specific example of the establishing of the virtual DSNaddress boundaries, the processing module establishes the virtual DSNaddress boundaries in accordance with boundaries of the data objects,where, for a first set of the data objects, the first storage unit ofthe set of the storage units stores an encoded data slice for each setof encoded data slices of the plurality of sets of encoded data slicescorresponding to the first set of the data objects and where, for asecond set of the data objects, the second storage unit of the set ofthe storage units stores an encoded data slice for each set of encodeddata slices of the plurality of sets of encoded data slicescorresponding to the second set of the data objects.

As yet another specific example of the establishing of the virtual DSNaddress boundaries, the processing module establishes the virtual DSNaddress boundaries in accordance with boundaries of the data segments ofthe data objects, where, for a first set of the data segments, the firststorage unit of the set of the storage units stores an encoded dataslice for each set of encoded data slices of the plurality of sets ofencoded data slices corresponding to the first set of the data segmentsand wherein, for a second set of the data segments, the second storageunit of the set of the storage units stores an encoded data slice foreach set of encoded data slices of the plurality of sets of encoded dataslices corresponding to the second set of the data segments.

The method continues at step 206 where the processing module detects achange in the given state of evolution of the DSN memory to a new givenstate when a change occurs to the virtual memory vault. When the changein the given state is detected, the method continues at step 208 wherethe processing module, for the new given state, interprets the trackinginformation for the new given state of evolution to produce new givenstate DSN memory data. The method continues at step 210 where theprocessing module establishes new virtual DSN address boundaries for theset of the storage units based on the new given state DSN memory data.

The method continues at step 212 where the processing module detects thechange in the given state of evolution of the DSN memory to the newgiven state when a change occurs to one of the storage units or when anew storage unit is added to the DSN memory. When the change in thegiven state is detected, the method continues at step 214 where theprocessing module, for the new given state, interprets the trackinginformation for the new given state of evolution to produce new givenstate DSN memory data. The method continues at step 216 where theprocessing module updates the set of storage units to produce an updatedset of storage units. The method continues at step 218 where theprocessing module establishes new virtual DSN address boundaries for theupdated set of the storage units based on the new given state DSN memorydata.

The method described above in conjunction with the DS managing unit 20can alternatively be performed by other modules of a dispersed storagenetwork or by other devices. In addition, at least one non-transitorycomputer readable storage medium that stores instructions that can, whenexecuted by one or more processing modules of one or more computingdevices of the dispersed storage network (DSN), cause the one or morecomputing devices to perform any or all of the method steps describedabove.

FIG. 6 a is a diagram of an embodiment of a dispersed storage addressmapping that includes a circular address sequence that begins and endsat the top with a first DSN address (e.g., 0) and a last DSN address(e.g., 256⁴⁸). The circular view of the entire DSN address rangefacilitates observation of the distancing relationships between theassigned address ranges, such as pillars of a particular vault. In anexample, a four pillar assignment with equal distance between pillaraddress ranges forms a pinwheel shape in the diagram. Equally spacingthe pillar address ranges apart provides an ability to expand eachportion in a commensurate way while simplifying the determination of theaddress ranges of each pillar.

Within a pillar range, the most significant bits of the DSN address areheld constant, including the slice index, the vault identifier, and thevault generation. Within a pillar range, the DSN address sequencesthrough the addresses to specify vault specific information includingthe file identifier or block identifier. The fields of the addressranges will be described in greater detail with reference to FIGS. 8 and11.

DS storage units are assigned pieces or all of the address range of asingle pillar. Multiple DS storage units may be assigned one pillar. DSstorage units serving the same pillar may be co-located at one site tobetter facilitate load balancing without utilizing network bandwidth. DSstorage units may or may not be assigned to more than one pillar.

Adding vault generations may serve to expand the address range of avault. Generations are discrete pools of storage that can be added to anexisting vault to increase its capacity. For example, the address rangefor a first generation may be assigned to one DS storage unit and theaddress range for a second generation may be assigned to another DSstorage unit. Such partitioning facilitates expansion without requiringfile transfers between DS storage units to balance the load. Theassignment and use of generations will be described in greater detailwith reference to FIGS. 14-17.

FIG. 6 b is a diagram of another embodiment of a dispersed storageaddress mapping that includes a close-in examination of one section ofthe DSN address circle of FIG. 6 a. In an example, vault λ has a pillar0 DSN address range very close to the vault θ pillar 0 DSN addressrange. Similar pillar numbers will be located near each other on thecircular DSN address circle since the slice index is the mostsignificant part of the DSN address and the slice index is derived fromthe pillar number. A single DS storage unit may be assigned to the samepillar number of two or more vaults. The determination of the sliceindex will be described in greater detail with reference to FIGS. 8-10.

FIG. 6 c is a diagram of another embodiment of a dispersed storageaddress mapping includes a circular address sequence that begins andends at the top with a first DSN address (e.g., 0) and a last DSNaddress (e.g., 256⁴⁸). The circular view of the entire DSN address rangefacilitates observation of the distancing relationships between theassigned address ranges, such as pillars of a particular vault. In anexample, an eight pillar assignment with equal distance between pillaraddress ranges forms a pinwheel shape in the diagram. Adding morepillars splits the space between the ranges. Equally spacing the pillaraddress ranges apart provides an ability to expand each portion in acommensurate way while simplifying the determination of the addressranges of each pillar.

Within a pillar range, the most significant bits of the DSN address areheld constant, including the slice index, the vault identifier, and thevault generation. Within a pillar range, the DSN address sequencesthrough the addresses to specify vault specific information includingthe file identifier or block identifier. The fields of the addressranges will be described in greater detail with reference to FIGS. 8 and11.

DS storage units are assigned pieces or all of the address range of asingle pillar. Multiple DS storage units may be assigned one pillar. DSstorage units serving the same pillar may be co-located at one site tobetter facilitate load balancing without utilizing network bandwidth. DSstorage units may or may not be assigned to more than one pillar.

FIG. 7 is a logic diagram of an embodiment of a method for storage of adata object within a dispersed storage network (DSN) that starts withthe step of slicing a data segment of a data object into data slices101. The slicing may be made by the processing module of the DSprocessing unit or of a user device.

The processing module next creates a slice name for each data slice 103.The slice name includes an identifier of the data object and a virtualmemory address of a virtual memory associated with the dispersed storagenetwork. For example, the slice name can include a file name/blockidentifier that identifies the data object and vault, data segment andslice information that provide the virtual memory address. The slicenames will be used as the virtual index to the memory system of each DSstorage unit to gain access the physical address of the EC data slices.

The processing module next outputs each of the data slices and therespective slice names to a corresponding storage unit for storage ofthe data slices therein 105. In an embodiment, each data slice, for thatdata segment, is stored on a different DS storage unit from all theother slices of that data segment to improve data integrity. The DSstorage units maintain respective local tables correlating slice names(virtual DSN address) to the addresses of the physical media internal tothe DS storage unit.

FIG. 8 is a diagram of an embodiment of a dispersed storage addresstable for the slice name 45 that includes routing information 91 uniqueacross the entire DSN and vault specific information 93 that containsvault specific source data name 95 that is unique to the vault user andmay be reused between vaults (e.g., the same file identifiers may beused in different vaults). The vault specific source data name 95 mayinclude file and segment identifiers or block identifiers 97 thatidentify the data object to be stored. The vault specific informationfields 97 will be further discussed with reference to FIG. 11.

The routing information field 91 may contain the slice index 65, thevault identifier 55, the vault generation 75, and the reserved field 99.In an embodiment the slice index 65 may be two bytes, the vaultidentifier 55 may be sixteen bytes, the vault generation 75 may be twobytes, and the reserved field 99 may be four bytes.

In an example, the user vault specifies that four pillars are used. Theslice index field is based on the vault ID and pillar ID such that thevery most significant bits sequence from 00 to 11 to provide the evenlydistributed DSN address ranges illustrated in FIG. 5. The determinationof the slice index will be further discussed with reference to FIG. 10.

The vault identifier may be a sixteen byte universally unique identifier(UUI). A given vault identifier is uniquely associated with one user onthe entire DSN and the vault identifier can be randomly assigned (butconstantly associated with the same user) by the DS processing unit, theDS managing unit, the storage integrity processing unit, and/or the DSstorage unit.

The vault generation may extend the address block for a given vault andfacilitate a simplified expansion process. The vault generation may beassigned by the DS processing unit, the DS managing unit, the storageintegrity processing unit, and/or the DS storage unit as will bediscussed further with reference to FIGS. 14-17.

FIG. 9 is a schematic block diagram of an embodiment of a dispersedstorage (DS) processing unit 102 that includes an interface 104, agateway module 106, an access module 108, a grid module 110, a storagemodule 112, and the DSnet interface 114. In another embodiment, one ormore of the gateway module 106, the access module 108, the grid module110, and the storage module 112 may be implemented outside of the DSprocessing unit 102. The DS processing unit 102 functions includes thetransformation of incoming data objects 122 (e.g., files, blocks,streams) from a user device 116 via the interface 104 into error coded(EC) data slices, with attached slice names, which the DS processingunit 102 stores in the DSN memory via the DSnet interface 114. The DSprocessing unit 102 retrieves EC data slices, with attached slice names,from the DSN memory and transforms the EC data slices into data objects,which the DS processing unit 102 communicates to the user device 116 viathe interface 104.

The interface 104 of the DS processing unit 102 is coupled to theinterface 126 of the user device 116. The interfaces 104, 126 may belogical if the DS processing unit 102 is part of the second user device.The interfaces 104, 126 collectively form a DSN memory interface toaccommodate the OS file system of the user device such that, from theperspective of the user device 116, data objects 122 may be stored toand retrieved from the DS processing unit 102 as if the DS processingunit 102 were any other compatible storage system. For example, theinterface may receive the data object 122, a user identifier 118 of theuser and an object name 120 (file name of the data object in the OS ofthe user) from interface 126 for storage of the data object.

The gateway module 106 couples the interface 104 to the access module108 and may include functions to act as a portal between the user device116 and the DS processing unit 102 allowing them to communicate betweenprotocols providing computing system interoperability. The gatewaymodule 106 converts protocols and messages as a function of the userdevice 116 OS file system. For example, the gateway module 106 convertsa retrieve-file message in the NFS protocol format from the first userdevice 116 to a dispersed storage network message in the dispersedstorage network protocol to trigger the DS processing unit 102 toretrieve the data and return it to the first user device 116. Thegateway module 106 may include other functions including access controlto prevent unauthorized use, user identification, user informationretrieval, traffic monitoring, statistics generation, DS processing unitconfiguration, and DS processing unit management. The gateway module 106may provide user access authentication to verify a user has permissionsto perform one or more DSN memory access functions including write,read, delete, list, status, configure, and/or other functions thataccess the DSN memory.

The gateway module 106 may access user information 115 based on the useridentification 118. The gateway module 106 receives the user identifier118 and looks up the vault identifier. Each vault is associated witheach user, and as described above, may contain user information 115,such as user attributes (e.g., who it is, billing data, etc.) andoperational parameters. The operational parameters may include one ormore of the error coding algorithm, the width X (number of pillars orslices per segment for this vault), the threshold T (described below),the encryption algorithm, a slicing parameter, a compression algorithm,an integrity check method, caching settings, parallelism settings,and/or other parameters that may be used to access the DSN memory. Thegateway module 106 may access user information 115 from the vaultassociated with the user from any one or more sources including the DSmanaging unit, the user device 116, the DSN memory, and/or a sourceexternal to the computing network.

The gateway module 106 may also determine a source name 121 to associatewith the data object 122. The source name 121 may contain a data name129 (block number or a file number) that is associated with the dataobject name 120, the vault generation number 125, the reserved field127, and the vault identifier 123. The data name 129 may be randomlyassigned but it is always associated with the user data object 122(object name 120). The gateway module 106 may pass the user information115, source name 121, and data object 122 to other elements of the DSprocessing unit 102 to assist in the operation as will be described ingreater detail with reference to FIGS. 10-17.

In an embodiment, the access module 108 communicates data objects 122,user information 115, and the source name 121 with the gateway module106, and data segments 117, user information 115, and the source name121 with the grid module 110. The access module 108 creates a series ofdata segments 1 through Y from the data object 122 to be stored. In oneembodiment, the number of segments Y is chosen (e.g., part of the userinformation) to be a fixed number for a given user vault, but the sizeof the segments varies as a function of the size of the data object 122.For instance, if the data object 122 is an image file of 4,194,304 eightbit bytes (e.g., 33,554,432 bits) and the number of segments Y=131,072,then each segment is 256 bits or 32 bytes. In another embodiment, thenumber of segments Y is a variable number based on the size of each datafile with the objective to have a constant size of the data segments(e.g., data segment size is specified in the user information). Forinstance, if the data object 122 is an image file of 4,194,304 bytes andthe fixed size of each segment is 4,096 bytes, the then number ofsegments Y=1,024.

The access module 108 may include the source name 121 in each the datasegment 117 before passing the data segment 117 to the grid module 110.The grid module 110 may pre-manipulate (e.g., compression, encryption,cyclic redundancy check (CRC), etc.) the data segment 117 beforecreating X error coded data slices 119 for each data segment 117. Thegrid module 110 creates XY error coded data slices 119 for the Y datasegments of the data object 122. The grid module 110 adds forward errorcorrection bits to the data segment bits in accordance with an errorcoding algorithm (e.g., Reed-Solomon, Convolution encoding, Trellisencoding, etc.) to produce an encoded data segment. The grid module 110determines the slice name 111 and attaches the unique slice name 111 toeach EC data slice 119.

The number of pillars, or slices X per data segment (e.g., X=16) ischosen as a function of the error coding objectives. A read threshold T(e.g., T=10) of the error coding algorithm is the minimum number oferror-free error coded data slices required to be able to reconstruct adata segment. The DS processing unit 102 can compensate for X−T (e.g.,16−10=6) missing error coded data slices per data segment.

The grid module 110 slices the encoded data segment 117 to produce theerror coded data slices 119. The slicing complements the error codingalgorithm and accommodates the nature of dispersing error coded dataslices to different DS storage units of the DSN memory. The grid module110 may use interleaving to slice the encoded data segment such that ifan entire error coded slice is lost it will not destroy a largecontiguous portion of the data segment preventing reconstruction.

The size of each error coded data slice is a function of the size of thedata segment and the error coding algorithm. In an embodiment, the sizeof each error coded data slice is the size of the data segment dividedby the threshold T. For example, if the data segment is 32 bytes and thethreshold is 10, then each error coded slice (without any addressing) isabout 4 bytes.

The grid module 110 may perform post-data manipulation on the errorcoded data slices where the manipulation may include one or more ofslice level compression, encryption, CRC, addressing, tagging, and/orother manipulation to improve the effectiveness of the computing system.

The grid module 110 may determine which DS storage units to store eacherror coded data slice based in part on a dispersed storage memorymapping associated with the user's vault as was described in withreference to FIGS. 3-4. The location determination may also be based inpart on DS storage unit attributes including one or more ofavailability, self-selection, performance history, link speed, linklatency, ownership, available memory, domain, cost, and more generallyon one or more of a prioritization scheme, a centralized selectionmessage from another source, a lookup table, who owns the data, and/orany other factor to optimize the operation of the computing system. Thegrid module 110 will pass the DS storage unit determinations on to thestorage module 112 so that the storage module 112 can direct the EC dataslices properly.

In an embodiment, the number of DS storage units is equal to or greaterthan the number of pillars (slices X per segment) so that no more thanone error coded data slice of the same data segment is stored on thesame DS storage unit. Error coded data slices of the same slice numberbut of different segments (e.g., EC data slice 1 of data segment 1 andEC data slice 1 of data segment 2) may be stored on the same ordifferent DS storage units.

The grid module 110 will pass the EC coded data slices 119 (includingthe slice names 111) with DS storage unit determinations (e.g., the DSstorage unit numbers) to the storage module 112. The storage module 112may perform integrity checks on the EC data slices and then transmit theEC data slices 1 through X of each segment 1 through Y to the DS Storageunits according to the previous location determination via the DSnetinterface 114. The DS storage units will store the EC data slices andlocally keep a table to convert virtual DSN addresses into physicalstorage addresses.

In an example of a data object retrieval operation, user device 116sends a data file retrieval message to the DS gateway module 106 of theDS processing unit 102. The gateway module 106 translates the data fileretrieval from the conventional file system interface into a retrievalmessage of a DSN memory interface convention. The grid module 110receives the retrieval message and determines where (e.g., which DSstorage units) the EC data slices for each segment of the data fileshould be. The storage module 112 retrieves the EC data slices from eachof the determined DS storage units and passes the retrieved EC dataslices to the grid module 110. The storage module may only retrieve thethreshold T number of EC data slices if they are known to be intact(e.g., no CRC errors etc. as indicated by the DS storage unit and/or bythe grid module 110). The grid module 110 de-slices the EC data slicesand decodes the de-sliced data in accordance with the error codingalgorithm to produce the data segments. The grid module 110 passes thedata segments to the access module 108, which re-assembles the data fileby aggregating the data segments in order. The access module 108 passesthe data file to the gateway module 106, which converts the format tothe file system protocol of the first user device 116.

FIG. 10 is a schematic block diagram of an embodiment of a grid module128 that includes a slice index generator 130. The slice index generator130 includes an exclusive OR (XOR) function 132 and a bit order reverser134. The bit order reverser 134 creates a reversed pillar 136 from thepillar identifier. The reverser 134 swaps the most significant bit withthe least significant bit, followed by reversing the second mostsignificant bit with the second least significant bit, followed by acontinuation of this pattern until all of the bits have been swapped.The reversed pillar 136 may contain the same number of bits as the sliceindex. If needed, the bit order reverser pads out zeros to fill thefield to the right.

The slice index generator 130 generates the slice index by performingthe XOR function 132 on the reversed pillar 136 and the leastsignificant bits of vault identifier 138. The XOR 132 process results inbetter distribution properties of addresses.

FIG. 11 is a diagram of another embodiment of a dispersed storageaddress table for the slice name 45 that includes routing information 91unique across the entire DSN and vault specific information 93 thatcontains vault specific source data name 95 that is unique to the vaultuser and may be reused between vaults (e.g., the same file identifiersmay be used in different vaults). The vault specific source data name 95may include file and segment identifiers or block identifiers.

The source data name 95 has at least two format instances. In a firstinstance for data files, the source data name 95 contains a type field131, the file identifier 133, and the data segment identifier 135. Thetype field 131 may be two bytes, the file identifier 133 may be sixteenbytes, and the data segment identifier 135 may be six bytes. In a secondinstance for data blocks, the source data name 95 contains a type field137, a filler field of zeros 139, and the block identifier 141. The typefield 137 has the same number of bytes as in the first instance, thefiller field 139 is fourteen bytes of zeros, and the block identifier141 is eight bytes.

The gateway module of FIG. 9 may set the type field based on the type ofdata object, e.g., file or blocks. The type field may also specify othertypes such as various live media streams, which may be stored as files,blocks, and/or another format such as a combination. The gateway modulemay assign the file identifier as a random number or based in part onthe object name (e.g., containing at least some of the bits from theobject name or as a result of an algorithm that utilizes the objectname). From that point forward, the object name and file identifier arealways linked and the relationship may be stored in the user vault.

The gateway module may assign the block identifier to be the same as theobject name (user OS system block ID) when the data object is a seriesof data blocks. The block identifier may also be set as a function of analgorithm that is based in part on the object name. In an example, thealgorithm may be an offset number translation (e.g., add 1000).

The DS processing module and all other DSN elements may read the typefield to determine how to interpret the rest of the source data name(e.g., files or blocks).

FIG. 12 a is a diagram of another embodiment of a dispersed storagememory mapping at time t1 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, vault λ hasa pillar 0 DSN address range very close to the vault θ pillar 0 DSNaddress range. Similar pillar numbers will be located near each other onthe circular DSN address circle since the slice index is the mostsignificant part of the DSN address and the slice index is derived fromthe pillar number. In an example, the DS processing unit may assign asingle DS storage unit A to the same pillar number of vaults λ and θ.Other storage units (not shown) may support the other pillars (notshown) of these vaults. In one instance, a single DS storage unit maysupport other same pillar number pillars of these vaults. In anotherinstance, different DS storage units may support other same pillarnumber pillars of these vaults. The assignment of DS storage units tovault address ranges may be done by the DS processing unit, the DSmanaging unit, the storage integrity processing unit, and/or the DSstorage unit. The DS processing unit may update the virtual DSN addressto physical DS storage unit table to indicate the above changes.

FIG. 12 b is a diagram of another embodiment of a dispersed storagememory mapping at time t2 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, vault λ hasa pillar 0 DSN address range close to the vault θ pillar 0 DSN addressrange. A vault α pillar 0 DSN address range has been assigned at t2. Inan example, the DS processing unit may assign a single DS storage unit Ato the same pillar number of vaults λ, θ, and now α since DS storageunit A has ample storage capacity at t2. Other storage units (not shown)may support the other pillars (not shown) of these vaults. In oneinstance, a single DS storage unit may support other same pillar numberpillars of these vaults. In another instance, different DS storage unitsmay support other same pillar number pillars of these vaults. The DSprocessing unit may update the virtual DSN address to physical DSstorage unit table to indicate the above changes.

FIG. 12 c is a diagram of another embodiment of a dispersed storagememory mapping at time t3 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, vault λ hasa pillar 0 DSN address range very close to the vault θ pillar 0 DSNaddress range and vault α pillar 0 DSN address range. The DS processingunit may assign vault β pillar 0 DSN address range at t3. In an example,the DS processing unit may assign a single DS storage unit A to the samepillar number of vaults λ, θ, α, and now β since DS storage unit A hasample storage capacity at t3. Other storage units (not shown) maysupport the other pillars (not shown) of these vaults. In one instance,a single DS storage unit may support other same pillar number pillars ofthese vaults. In another instance, different DS storage units maysupport other same pillar number pillars of these vaults. The DSprocessing unit may update the virtual DSN address to physical DSstorage unit table to indicate the above changes.

FIG. 12 d is a diagram of another embodiment of a dispersed storagememory mapping at time t4 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, vault λ hasa pillar 0 DSN address range very close to the vault θ pillar 0 DSNaddress range, and the vault α has a pillar 0 DSN address range veryclose to the vault β pillar 0 DSN address range. DS storage unit B isadded at the time t4 to carry some of the storage load. DS storage unitB may be collocated at the same site as DS storage unit A since theywill serve the same pillar numbers and co-location will facilitate afaster transfer of EC data slices from DS storage unit A to DS storageunit B to balance the load. In an example, the DS processing unit mayassign DS storage unit A to pillar number 0 of vaults λ, and θ, while DSstorage unit B will be assigned to pillar 0 of vaults α and β. In thatassignment, the DS processing unit may facilitate the transfer of thepillar 0 EC data slices for vaults x and y to DS storage unit B. Otherstorage units (not shown) may support the other pillars (not shown) ofthese vaults. In one instance, a single DS storage unit may supportother same pillar number pillars of these vaults. In another instance,different DS storage units may support other same pillar number pillarsof these vaults. The re-assignment of DS storage units to vault addressranges and the re-balancing of EC data slices may be done by the DSprocessing unit, the DS managing unit, the storage integrity processingunit, and/or the DS storage unit. The DS processing unit may update thevirtual DSN address to physical DS storage unit table to indicate theabove changes.

FIG. 12 e is a diagram of another embodiment of a dispersed storagememory mapping at time t5 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, vault λ hasa pillar 0 DSN address range close to the vault θ pillar 0 DSN addressrange, and the vault α has a pillar 0 DSN address range very close tothe vault β pillar 0 DSN address range. DS storage unit C is added atthe time t5 to support new vaults φ and μ, which have similar addressranges for pillar 0. DS storage unit C may be collocated at the samesite as DS storage units A and B since they will serve the same pillarnumbers and co-location will facilitate any required future transfer ofEC data to balance the load. In an example, the DS processing unit mayassign DS storage unit A to pillar number 0 of vaults λ, and θ, and DSstorage unit B will be assigned to pillar 0 of vaults α and β, while DSstorage unit C will be assigned to pillar 0 of vaults φ and μ. Otherstorage units (not shown) may support the other pillars (not shown) ofthese vaults. In one instance, a single DS storage unit may supportother same pillar number pillars of these vaults. In another instance,different DS storage units may support other same pillar number pillarsof these vaults. The DS processing unit may update the virtual DSNaddress to physical DS storage unit table to indicate the above changes.

FIG. 12 f is a diagram of another embodiment of a dispersed storagememory mapping at time t6 that includes a close-in examination of onesection of the DSN address circle of FIG. 5. In an example, the DSprocessing module determines that DS storage unit A is filling up whileDS storage unit C has ample capacity. The DS processing moduledetermines that vault λ pillar 0 will be transferred to DS storage unitC. The determination may be based on remaining capacity, performance,billing rates, user ownership, and or some other metric to driveimproved operations. Other storage units (not shown) may support theother pillars (not shown) of these vaults. In one instance, a single DSstorage unit may support other same pillar number pillars of thesevaults. In another instance, different DS storage units may supportother same pillar number pillars of these vaults. The DS processing unitmay update the virtual DSN address to physical DS storage unit table toindicate the above changes.

FIG. 13 is a logic diagram of an embodiment of a method for addressassignment that starts with the step of determining if a new DS storageunit 140 has been added to the DSN. The determination may be made by thereceipt of a registration message by the processing module of any of theDS managing unit, the DS processing unit, the storage integrityprocessing unit, and/or another DS storage unit.

The processing module determines the site location of the new DS storageunit 142. The determination may be indicated by the registration messageor it may be indicated by a proximity message from the new DS storageunit. The new DS storage unit may determine which other DS storage unitsare at the same site by performing a series of local area network (LAN)pings. The new DS storage unit will create the proximity message basedon the results of the LAN pings. The DS storage unit may also accept asite indicator input from a configuration operation (e.g., from a localmanager or the DS managing unit).

DS storage units may be simultaneously added as a unit set to multiplesites to support each pillar of one or more vaults. The processingmodule site determination may also determine if a unit set has beenactivated by correlating the new sites to the virtual DSN address tophysical DS storage unit tables (as discussed in FIG. 4).

The processing module determines the loading for existing DS storageunits at the site(s) indicated by the site determination for the new DSstorage units 144. The loading may indicate available storage capacityand/or performance metrics that affect loading including storage latencytimes, retrieval latency times, network access bandwidth, availability,memory usage growth rate (e.g., bytes of incremental storage used perunit of time), and/or any other past performance metric that mayindicate potential future performance. Available storage capacity may beexpressed as bytes or as a percentage of the total size of the DSstorage unit storage capacity.

The processing module determines which existing DS storage unit(s) toadjust by comparing the loading to a loading threshold 146. In anexample, the loading threshold may indicate that DS storage units shouldbe adjusted if the available storage capacity falls below 10% of thetotal. In another example, the loading threshold may indicate that DSstorage units should be adjusted if the available storage capacity ispredicted to only last less than one week based on the current memoryusage growth rate. The processing module may attempt to match a new unitset to an existing unit set of DS storage units that have the mostunfavorable loading as compared to the loading thresholds. Theprocessing module may match DS storage units and new DS storage units atthe same site to facilitate rapid file transfer in the rebalancing stepover a LAN connection.

The processing module determines what volume and which particular ECdata slices to move from the DS storage unit to the new DS storage unit148. Volume determination may be based on a threshold, a predeterminednumber, a DS managing unit input, and/or a target new percentage ofavailable storage capacity. For example, the new percentage of availablestorage capacity may be 60% such that the DS storage unit will be 40%full after the rebalancing step. The determination of which EC dataslices to move may be based on which user vaults are present, vaultboundaries, generation boundaries, and/or file boundaries. In anexample, vault λ pillar 0, generations 0000-7FFF will remain on the DSstorage unit but generations >7F00 will be moved to the new DS storageunit. As generations may be quite large, portions of the same generationmay be split between two or more DS storage units. In such a scenario,the split may occur on file boundaries such that all of the same pillarsegments for the same data file are stored on the same DS storage unit.

The processing module updates the virtual DSN address to physical DSstorage unit tables to reflect the new configuration 150. Subsequentstore and retrieve operations will utilize the updated information toaccess the appropriate DS storage units.

FIG. 14 is a diagram of another embodiment of a dispersed storageaddress table for the slice name 45 that includes routing information 91unique across the entire DSN and vault specific information 93 thatcontains vault specific source data name that is unique to the vaultuser per generation and may be reused between vaults (e.g., the samefile identifiers may be used in different vaults). The vault specificsource data name 95 may include file and segment identifiers or blockidentifiers. The vault specific information fields were discussed inmore detail above with reference to FIG. 11.

The routing information field 91 may contain the slice index 65, thevault identifier 55, the vault generation 75, and the reserved field 99.In an embodiment the slice index 65 may be two bytes, the vaultidentifier 55 may be a sixteen bytes, the vault generation 75 may be twobytes, and the reserved field 99 may be four bytes.

In an example, the user vault specifies that four pillars are used. Theslice index field 65 is based on the vault ID and pillar ID such thatthe very most significant bits sequence from 00 to 11 to provide theevenly distributed DSN address ranges illustrated in FIG. 5. Thedetermination of the slice index was discussed in more detail above withreference to FIG. 10.

The vault identifier 55 may be a sixteen byte universally uniqueidentifier (UUI). A given vault identifier is uniquely associated withone user on the entire DSN and the vault identifier can be randomlyassigned (but constantly associated with the same user) by the DSprocessing unit, the DS managing unit, the storage integrity processingunit, and/or the DS storage unit.

The vault generation 75 may extend the address block for a given vaultand facilitate a simplified expansion process. The vault generation maybe assigned by the DS processing unit, the DS managing unit, the storageintegrity processing unit, and/or the DS storage. In an example, thegrid module determines to add a new generation 1 to vault λ based on anew DS storage unit being added and/or if a threshold of generationutilization has been reached. The processing module of the grid modulemay add the new generation 1. The new generation may also be added bythe DS managing unit, the storage integrity processing unit, the userdevice, and/or the DS storage unit.

In an example, a new DS storage unit may be added to support the newgeneration 1. In such a scenario, the processing module updates thevirtual DSN address to physical DS storage unit table to reflect thatthe old DS storage unit for that particular vault supports an addressrange inclusive of generation 0 and that a new DS storage unit for thesame vault supports an address range inclusive of the new generation 1.

FIG. 15 a is a diagram of another embodiment of a dispersed storageaddress mapping that includes a close-in examination of one section ofthe DSN address circle of FIG. 5 focusing on the ranges of generationsof a given vault. In an example, where the vault generation field is twobytes, the vault generation numbers will range from 0000 to FFFF. Thegrid module may assign new vault generation numbers from time to time aseither new DS storage units are added or when current generations reacha capacity threshold. The vault generation number may be randomlyassigned within its range or it may be assigned according to analgorithm. The method of assignment of the vault generation number willbe further discussed with reference to FIGS. 15 b-15 c.

FIG. 15 b is a diagram of another embodiment of a dispersed storageaddress mapping where the vault generation range is represented around acircle. At the top are the first and the last numbers (e.g., 0000 andFFFF). The grid module may utilize an even distribution algorithm to addnew vault generations such that DS storage assignments may never supportadjacent vault generations to promote a more robust system. In otherwords, as DS storage units generally cover sequential ranges of virtualDSN addresses as illustrated in FIG. 4, avoid a potential simultaneousfailure of two or more adjacent generations by assigning vaults to vaultgenerations that have significant distance between them on the addresscircle. For example the first generation to be assigned is generation 0and it may be assigned a vault generation number of 0100 on the circle.As that generation fills and when a new DS storage unit is activated,the second generation to be assigned is generation 1 and it may beassigned a vault generation number of 8100. The even distancingalgorithm continues and subsequent generations may be added asillustrated in FIG. 15 b. DS storage units supporting sequential addressranges may eventually support more than one generation but they will notbe sequential in generation age. A DS storage unit failure will notaffect two generations of adjacent age.

FIG. 15 c is a diagram of another embodiment of a dispersed storageaddress mapping where the grid module utilizes a sequential vaultgeneration number assignment algorithm to add new generations. Forexample, the grid module assigns the first generation to vaultgeneration number 0000 and the grid module assigns the second generationto vault generation number 0001 and so on in sequence. The grid modulemay assign new DS storage units to each new vault generation such that aDS storage unit failure will only impact one generation.

FIG. 16 a is a logic diagram of another embodiment of a method foraddress assignment where the processing module determines that a newgeneration should be added for a vault 161. The determination may bebased on, for example, a comparison of the storage utilization withinthe physical memory of a current generation of the vault to a generationutilization threshold. When the storage utilization compares unfavorablyto the generation utilization threshold, the new generation is added.The processing module may be a part of the DS managing unit, the DSstorage unit, the storage integrity processing unit, the DS processingunit and/or the user device.

The processing module assigns a vault generation identifier to the newgeneration that uniquely identifies the new generation within thedispersed storage network 163. In addition, the processing moduleassigns a virtual address range within the virtual memory to the newgeneration 165. For example, the processing module can randomly assignthe virtual address range to the new generation, utilize an evendistribution algorithm to assign the virtual address range to the newgeneration or utilize a sequential assignment algorithm to assign thevirtual address range to the new generation.

The processing module further maps the assigned virtual address range tophysical memory 167. For example, since the physical memory includes aplurality of storage units, the processing module can update a tablemapping the respective virtual address range of each generation of thevault to corresponding storage units. In an exemplary embodiment, thedata associated with the new generation is stored in different storageunits than the data associated with previous generations of the vault.

FIG. 16 b is a logic diagram of another embodiment of a method foraddress assignment where, at step 152, a processing module (e.g., of thedispersed storage (DS) managing unit 20 of FIG. 1, the DS unit 36 ofFIG. 1, the DS processing unit 16 of FIG. 1, the user device 12 ofFIG. 1) detects a new DS storage unit within the dispersed storagenetwork (DSN) of FIG. 1. The detecting may include detecting a pluralityof new storage units within the DSN, where the plurality of new storageunits includes the new storage unit. The detecting may be based on oneor more of a registration message, a network ping, a manager command,and/or any other indicator that a new DS storage unit is now online.

The method continues at step 154 where the processing module determinesif the new DS storage unit has companions in a storage unit set (e.g.,at the same sites as pillars of at least one vault, the plurality ofnewly detected storage units includes the storage unit set including thenew storage unit). The method branches to step 162 when the processingmodule determines that the new storage unit is part of the storage unitset. When the DS storage unit is not part of the unit set, the methodcontinues at step 156 where the processing module determines which oldDS storage units (e.g., previously activated DS storage units) at thesite of the new DS storage unit to re-balance. For example, theprocessing module determines to affiliate the new storage unit with avirtual memory vault, where the virtual memory vault is mapped to a setof storage units of the DSN, where data objects are dispersed storageerror encoded into pluralities of sets of encoded data slices that arestored in the set of storage units. The determining may include at leastone of determining that available physical memory of one or more storageunits of the set of storage units is below a desired availabilitythreshold and receiving a request to allocate the DSN addresses of thevirtual memory to the new storage unit.

The method continues at step 158 where the processing module determinesvirtual DSN addresses of the virtual memory vault to allocate the newstorage unit to re-balance error coded data slices. The determining mayinclude ascertaining a range of virtual DSN addresses assigned to thevault, determining virtual DSN addresses of the range of virtual DSNaddresses that are unallocated to storage units of the set of storageunits to produce unallocated virtual DSN addresses, and determining toallocate at least some of the unallocated virtual DSN addresses to thenew storage unit as the virtual DSN addresses. Alternatively, thedetermining may include determining virtual DSN addresses that arecurrently allocated to a storage unit of the set of storage units toproduce allocated virtual DSN addresses and reallocating some of theallocated virtual DSN addresses as the virtual DSN addresses to the newstorage unit. For example, the processing module chooses another DSstorage unit with a least remaining storage capacity (e.g., determiningthat available physical memory of one or more storage devices of the setof storage devices is below a desired availability threshold) or yetanother DS storage unit with another loading performance issue (e.g.,transactions are slower than an average transaction execution time frameof other DS storage units). Having determined the virtual DSN addressesof the virtual memory vault to allocate to the new storage unit, theprocessing module transfers error coded (EC) data slices from theselected old DS storage unit to the new DS storage unit.

The method continues at step 160 where the processing module updatesvault information regarding the virtual memory vault to include theallocation of the virtual DSN addresses to the new storage unit. Forexample, the processing module updates virtual DSN address range tophysical DS storage tables. The processing module may further determinean amount of the EC data slices to transfer based on a target capacitythreshold after the transfer. For instance, the desired storage capacitythreshold may be 70% available after the transfer. The updating mayfurther include sending a vault affiliation message to the new storageunit regarding the assignment of the virtual DSN addresses, anaffiliation with the vault, and data access processing parameters (e.g.,dispersal parameters) for the vault.

Alternatively, the processing module determines to affiliate the newstorage unit with a second virtual memory vault, where the secondvirtual memory vault is mapped to a second set of storage units of theDSN. The second set of storage units may be mutually exclusive from thefirst set of storage units. The second set of storage units may furtherinclude one or more common storage units of the first set of storageunits. The processing module determines the second virtual DSN addressesof the second virtual memory vault to allocate the new storage unit andupdates the second vault information regarding the second virtual memoryvault to include the allocation of the second virtual DSN addresses tothe new storage unit.

When the new storage unit is part of the storage unit set (e.g., a newset of storage units has been detected), the method continues at step162 where the processing module determines whether a new generation isto be added. The determining may be based on comparing storageutilization within a current generation to a generation utilizationthreshold. For example, the processing module indicates not to add thenew generation when determining that a storage utilization of thecurrent generation is 5% and a generation utilization threshold for thisvault is 30%. The method branches to step 170 when the processing moduleindicates to add the new generation. The method continues to step 164when the processing module indicates not to add the new generation.

The method continues at steps 164-168 where the processing moduledetermines to affiliate at least some of the plurality of new storageunits with the virtual memory vault, determines sets of virtual DSNaddresses of the virtual memory vault to allocate the at least some ofthe plurality of the storage units, and updates the vault information toinclude the allocation of sets of virtual DSN addresses of the virtualmemory vault to allocate the at least some of the plurality of DSstorage units. For example, when the processing module determines thatthe new generation is not to be added, the processing module, at step164, chooses a current storage unit set that is closest to thegeneration utilization threshold (e.g., most full, but not so full as toadd a new generation). The method continues at step 166 where theprocessing module transfers EC data slices from each DS storage unit ofthe chosen old storage unit set to the new DS storage units of the newunit set. The method continues at step 168 where the processing moduleupdates the virtual DSN address range to physical DS storage tables. Theprocessing module may determine an amount of EC data slices to transferbased on a target capacity threshold after the transfer. For example,the desired storage capacity threshold may be 70% available after thetransfer.

The method continues at steps 170-172 where the processing moduledetermines to map at least some of the plurality of new storage units toa new virtual memory vault, determines a new virtual DSN address for thenew virtual memory vault, and creates new vault information regardingthe new virtual memory vault to include the allocation of new virtualDSN addresses to the at least some of the plurality of the storage unitswhen the processing module indicates that the new generation. Forexample, at step 170, the processing module determines which old storageunit set to add the new generation when the processing module determinesthat the new generation is to be added. For instance, the processingmodule may find an existing storage unit set with a high utilization(e.g., above a threshold) of the current generation. The methodcontinues at step 172 where the processing module adds the generationfor this storage unit set by updating the virtual DSN address tophysical DS storage unit tables with the new vault generation rangemapped to the set of DS storage units.

The method described above in conjunction with the DS managing unit 20can alternatively be performed by other modules of a dispersed storagenetwork or by other devices. In addition, at least one non-transitorycomputer readable storage medium that stores instructions that can, whenexecuted by one or more processing modules of one or more computingdevices of the dispersed storage network (DSN), cause the one or morecomputing devices to perform any or all of the method steps describedabove.

FIG. 17 is a logic diagram of another embodiment of a method for addressassignment where the processing module of the DS processing unitdetermines if multiple generations are available for a user vault when anew data object is to be dispersed 174. The processing module may bepart of the DS managing unit, the DS storage unit, the storage integrityprocessing unit, the DS processing unit, and/or the user device. Thedetermination may be based on a lookup into the virtual DSN address tophysical DS storage unit tables. The processing module may assign thecurrent vault generation when multiple generations are not available176. The processing module fills in the vault generation field with thesame current generation in the slice name for each dispersed EC dataslice.

The processing module may determine the generation utilization levelswhen multiple generations are available 178. The determination may bebased on storage utilization for each generation range (e.g., how muchbulk data is stored for each generation) and/or source data nameutilization (e.g., file identifiers). In an embodiment, the processingmodule may list each generation from oldest to newest with the amount ofdata stored and the number of source data names used for each.

The processing module may determine if the oldest generation hasutilization levels greater than a first utilization threshold 180. Forexample, the amount of data stored in the generation may be over 10terabytes and the number of source names utilized may be above 405 ofthe total possible numbers. If either is above its threshold, then thegeneration is above the threshold. The processing module may assign theoldest generation when the oldest generation has utilization levels thatare not greater than the first utilization threshold 182.

The processing module may determine if the oldest generation hasutilization levels greater than a second utilization threshold when theprocessing module has determined that the oldest generation hasutilization levels greater than a first utilization threshold 184. Theprocessing module may determine if the next oldest generation hasutilization levels greater than the first utilization threshold when theprocessing module has determined that the oldest generation does nothave utilization levels greater than the second utilization threshold186. The processing module may assign the next oldest generation whenthe next oldest generation does not have utilization levels greater thanthe first utilization threshold 188. The processing module may assignthe oldest generation when the next oldest generation has utilizationlevels greater than the first utilization threshold 190.

The processing module may determine if the next oldest generation hasutilization levels greater than the second utilization threshold whenthe processing module has determined that the oldest generation hasutilization levels greater than the second utilization threshold 192.The processing module may assign the next oldest generation when thenext oldest generation does not have utilization levels greater than thesecond utilization threshold 194. The processing module may continue theprocess to the next oldest generation when the next oldest generationhas utilization levels greater than the second utilization threshold196. The process continues until the either the generation is assignedor the current generation is assigned.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “coupled to” and/or “coupling” and/or includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for indirect coupling, theintervening item does not modify the information of a signal but mayadjust its current level, voltage level, and/or power level. As mayfurther be used herein, inferred coupling (i.e., where one element iscoupled to another element by inference) includes direct and indirectcoupling between two items in the same manner as “coupled to”. As mayeven further be used herein, the term “operable to” indicates that anitem includes one or more of power connections, input(s), output(s),etc., to perform one or more its corresponding functions and may furtherinclude inferred coupling to one or more other items. As may stillfurther be used herein, the term “associated with”, includes directand/or indirect coupling of separate items and/or one item beingembedded within another item. As may be used herein, the term “comparesfavorably”, indicates that a comparison between two or more items,signals, etc., provides a desired relationship. For example, when thedesired relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1.

The present invention has also been described above with the aid ofmethod steps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claimed invention.

The present invention has been described above with the aid offunctional building blocks illustrating the performance of certainsignificant functions. The boundaries of these functional buildingblocks have been arbitrarily defined for convenience of description.Alternate boundaries could be defined as long as the certain significantfunctions are appropriately performed. Similarly, flow diagram blocksmay also have been arbitrarily defined herein to illustrate certainsignificant functionality. To the extent used, the flow diagram blockboundaries and sequence could have been defined otherwise and stillperform the certain significant functionality. Such alternatedefinitions of both functional building blocks and flow diagram blocksand sequences are thus within the scope and spirit of the claimedinvention. One of average skill in the art will also recognize that thefunctional building blocks, and other illustrative blocks, modules andcomponents herein, can be implemented as illustrated or by discretecomponents, application specific integrated circuits, processorsexecuting appropriate software and the like or any combination thereof.

What is claimed is:
 1. A method for execution by one or more processingmodules of one or more computing devices of a dispersed storage network(DSN), the method comprises: detecting a new storage unit within theDSN; determining to affiliate the new storage unit with a virtual memoryvault, wherein the virtual memory vault is mapped to a set of storageunits of the DSN, wherein data objects are dispersed storage errorencoded into pluralities of sets of encoded data slices that are storedin the set of storage units; determining virtual DSN addresses of thevirtual memory vault to allocate the new storage unit; and updatingvault information regarding the virtual memory vault to include theallocation of the virtual DSN addresses to the new storage unit.
 2. Themethod of claim 1, wherein the determining to affiliate the new storageunit with the virtual memory vault comprises at least one of:determining that available physical memory of one or more storage unitsof the set of storage units is below a desired availability threshold;and receiving a request to allocate the DSN addresses of the virtualmemory vault to the new storage unit.
 3. The method of claim 1, whereinthe determining the virtual DSN addresses of the virtual memory vault toallocate the new storage unit comprises: ascertaining a range of virtualDSN addresses assigned to the vault; determining virtual DSN addressesof the range of virtual DSN addresses that are unallocated to storageunits of the set of storage units to produce unallocated virtual DSNaddresses; and determining to allocate a least some of the unallocatedvirtual DSN addresses to the new storage unit as the virtual DSNaddresses.
 4. The method of claim 1 further comprises: sending a vaultaffiliation message to the new storage unit regarding assignment of thevirtual DSN addresses, an affiliation with the vault, and data accessprocessing parameters for the vault.
 5. The method of claim 1 whereinthe determining the virtual DSN addresses of the virtual memory vault toallocate the new storage unit comprises: determining virtual DSNaddresses that are currently allocated to a storage unit of the set ofstorage units to produce allocated virtual DSN addresses; andreallocating some of the allocated virtual DSN addresses as the virtualDSN addresses to the new storage unit.
 6. The method of claim 1 furthercomprises: determining to affiliate the new storage unit with a secondvirtual memory vault, wherein the second virtual memory vault is mappedto a second set of storage units of the DSN; determining second virtualDSN addresses of the second virtual memory vault to allocate the newstorage unit; and updating second vault information regarding the secondvirtual memory vault to include the allocation of the second virtual DSNaddresses to the new storage unit.
 7. The method of claim 1 furthercomprises: detecting a plurality of new storage units within the DSN,wherein the plurality of new storage units includes the new storageunit; determining to map at least some of the plurality of new storageunits to a new virtual memory vault; determining new virtual DSNaddresses for the new virtual memory vault; and creating new vaultinformation regarding the new virtual memory vault to include theallocation of the new virtual DSN addresses to the at least some of theplurality of new storage units.
 8. The method of claim 1 furthercomprises: detecting a plurality of new storage units within the DSN,wherein the plurality of new storage units includes the new storageunit; determining to affiliate at least some of the plurality of newstorage units with the virtual memory vault; determining sets of virtualDSN addresses of the virtual memory vault to allocate the at least someof the plurality of new storage units; and updating the vaultinformation to include the allocation of the sets of virtual DSNaddresses of the virtual memory vault to allocate the at least some ofthe plurality of new storage units.
 9. A non-transitory computerreadable storage medium having accessible therefrom a set ofinstructions interpretable by a processing module, the set ofinstructions being configured to cause the processing module to carryout operations for: detecting a new storage unit within a dispersedstorage network (DSN); determining to affiliate the new storage unitwith a virtual memory vault, wherein the virtual memory vault is mappeda set of storage units of the DSN, wherein data objects are dispersedstorage error encoded into pluralities of sets of encoded data slicesthat are stored in the set of storage units; determining virtual DSNaddresses of the virtual memory vault to allocate the new storage unit;and updating vault information regarding the virtual memory vault toinclude the allocation of the virtual DSN addresses to the new storageunit.
 10. The storage medium of claim 9, wherein the set of instructionsfurther causes the processing module to carry out operations for thedetermining to affiliate the new storage unit with the virtual memoryvault by at least one of: determining that available physical memory ofone or more storage units of the set of storage units is below a desiredavailability threshold; and receiving a request to allocate the DSNaddresses of the virtual memory vault to the new storage unit.
 11. Thestorage medium of claim 9, wherein the set of instructions furthercauses the processing module to carry out operations for the determiningthe virtual DSN addresses of the virtual memory vault to allocate thenew storage unit by: ascertaining a range of virtual DSN addressesassigned to the vault; determining virtual DSN addresses of the range ofvirtual DSN addresses that are unallocated to storage units of the setof storage units to produce unallocated virtual DSN addresses; anddetermining to allocate a least some of the unallocated virtual DSNaddresses to the new storage unit as the virtual DSN addresses.
 12. Thestorage medium of claim 9, wherein the set of instructions further beingconfigured to cause the processing module to carry out operations for:sending a vault affiliation message to the new storage unit regardingassignment of the virtual DSN addresses, an affiliation with the vault,and data access processing parameters for the vault.
 13. The storagemedium of claim 9, wherein the set of instructions further causes theprocessing module to carry out operations for the determining thevirtual DSN addresses of the virtual memory vault to allocate the newstorage unit by: determining virtual DSN addresses that are currentlyallocated to a storage unit of the set of storage units to produceallocated virtual DSN addresses; and reallocating some of the allocatedvirtual DSN addresses as the virtual DSN addresses to the new storageunit.
 14. The storage medium of claim 9, wherein the set of instructionsfurther being configured to cause the processing module to carry outoperations for: determining to affiliate the new storage unit with asecond virtual memory vault, wherein the second virtual memory vault ismapped to a second set of storage units of the DSN; determining secondvirtual DSN addresses of the second virtual memory vault to allocate thenew storage unit; and updating second vault information regarding thesecond virtual memory vault to include the allocation of the secondvirtual DSN addresses to the new storage unit.
 15. The storage medium ofclaim 9, wherein the set of instructions further being configured tocause the processing module to carry out operations for: detecting aplurality of new storage units within the DSN, wherein the plurality ofnew storage units includes the new storage unit; determining to map atleast some of the plurality of new storage units to a new virtual memoryvault; determining new virtual DSN addresses for the new virtual memoryvault; and creating new vault information regarding the new virtualmemory vault to include the allocation of the new virtual DSN addressesto the at least some of the plurality of new storage units.
 16. Thestorage medium of claim 9, wherein the set of instructions further beingconfigured to cause the processing module to carry out operations for:detecting a plurality of new storage units within the DSN, wherein theplurality of new storage units includes the new storage unit;determining to affiliate at least some of the plurality of new storageunits with the virtual memory vault; determining sets of virtual DSNaddresses of the virtual memory vault to allocate the at least some ofthe plurality of new storage units; and updating the vault informationto include the allocation of the sets of virtual DSN addresses of thevirtual memory vault to allocate the at least some of the plurality ofnew storage units.
 17. A computing device comprises: an interface;memory; and a processing module operably coupled to the interface andthe memory, wherein the processing module is operable to: detect a newstorage unit within a dispersed storage network (DSN); determine toaffiliate the new storage unit with a virtual memory vault, wherein thevirtual memory vault is mapped to a set of storage units of the DSN,wherein data objects are dispersed storage error encoded intopluralities of sets of encoded data slices that are stored in the set ofstorage units; determine virtual DSN addresses of the virtual memoryvault to allocate the new storage unit; and update vault informationregarding the virtual memory vault to include the allocation of thevirtual DSN addresses to the new storage unit.
 18. The computing deviceof claim 17, wherein the processing module functions to determine toaffiliate the new storage unit with the virtual memory vault by at leastone of: determining that available physical memory of one or morestorage units of the set of storage units is below a desiredavailability threshold; and receiving a request to allocate the DSNaddresses of the virtual memory vault to the new storage unit.
 19. Thecomputing device of claim 17, wherein the processing module functions todetermine the virtual DSN addresses of the virtual memory vault toallocate the new storage unit by: ascertaining a range of virtual DSNaddresses assigned to the vault; determining virtual DSN addresses ofthe range of virtual DSN addresses that are unallocated to storage unitsof the set of storage units to produce unallocated virtual DSNaddresses; and determining to allocate a least some of the unallocatedvirtual DSN addresses to the new storage unit as the virtual DSNaddresses.
 20. The computing device of claim 17, wherein the processingmodule further functions to: send a vault affiliation message to the newstorage unit regarding assignment of the virtual DSN addresses, anaffiliation with the vault, and data access processing parameters forthe vault.
 21. The computing device of claim 17, wherein the processingmodule functions to determine the virtual DSN addresses of the virtualmemory vault to allocate the new storage unit by: determining virtualDSN addresses that are currently allocated to a storage unit of the setof storage units to produce allocated virtual DSN addresses; andreallocating some of the allocated virtual DSN addresses as the virtualDSN addresses to the new storage unit.
 22. The computing device of claim17, wherein the processing module further functions to: determine toaffiliate the new storage unit with a second virtual memory vault,wherein the second virtual memory vault is mapped to a second set ofstorage units of the DSN; determine second virtual DSN addresses of thesecond virtual memory vault to allocate the new storage unit; and updatesecond vault information regarding the second virtual memory vault toinclude the allocation of the second virtual DSN addresses to the newstorage unit.
 23. The computing device of claim 17, wherein theprocessing module further functions to: detect a plurality of newstorage units within the DSN, wherein the plurality of new storage unitsincludes the new storage unit; determine to map at least some of theplurality of new storage units to a new virtual memory vault; determinenew virtual DSN addresses for the new virtual memory vault; and createnew vault information regarding the new virtual memory vault to includethe allocation of the new virtual DSN addresses to the at least some ofthe plurality of new storage units.
 24. The computing device of claim17, wherein the processing module further functions to: detect aplurality of new storage units within the DSN, wherein the plurality ofnew storage units includes the new storage unit; determine to affiliateat least some of the plurality of new storage units with the virtualmemory vault; determine sets of virtual DSN addresses of the virtualmemory vault to allocate the at least some of the plurality of newstorage units; and update the vault information to include theallocation of the sets of virtual DSN addresses of the virtual memoryvault to allocate the at least some of the plurality of new storageunits.